Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Julia template for binary string #20355

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

qiaojunfeng
Copy link

It seems there is a small issue with generated julia function signature, when handling

application/octet-stream:
  schema:
    type: string
    format: binary

To better reproduce this, I set up a MWE here
https://github.com/qiaojunfeng/test-julia-OpenAPI/tree/main

Since this is a julia-specific issue, I will ping technical committee member @tanmaykm. Thanks!

PR checklist

  • Read the contribution guidelines.
  • Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
  • Run the following to build the project and update samples:
    ./mvnw clean package || exit
    ./bin/generate-samples.sh ./bin/configs/*.yaml || exit
    ./bin/utils/export_docs_generators.sh || exit
    
    (For Windows users, please run the script in Git BASH)
    Commit all changed files.
    This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master.
    These must match the expectations made by your contribution.
    You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*.
    IMPORTANT: Do NOT purge/delete any folders/files (e.g. tests) when regenerating the samples as manually written tests may be removed.
  • File the PR against the correct branch: master (upcoming 7.x.0 minor release - breaking changes with fallbacks), 8.0.x (breaking changes without fallbacks)
  • If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request.

@wing328 wing328 added this to the 7.11.0 milestone Dec 21, 2024
@tanmaykm
Copy link
Contributor

Thanks @qiaojunfeng, I shall check this out this weekend.

@tanmaykm
Copy link
Contributor

The effect of this change would be that if something is specified as binary format, it would now be generated as a String instread of Vector{UInt8} bytearray. While strings in Julia can contain invalid UTF8 bytecodes (https://docs.julialang.org/en/v1/manual/strings/), canonically Vector{UInt8} is used for such purposes.

I would like to understand more the reason why you feel we should have this changed in the openapi generated code?

@qiaojunfeng
Copy link
Author

The effect of this change would be that if something is specified as binary format, it would now be generated as a String instread of Vector{UInt8} bytearray. While strings in Julia can contain invalid UTF8 bytecodes (https://docs.julialang.org/en/v1/manual/strings/), canonically Vector{UInt8} is used for such purposes.

I would like to understand more the reason why you feel we should have this changed in the openapi generated code?

Thanks @tanmaykm!
A concise answer to this is that with the current generator, the generated julia code does not work with binary string for file upload. Probably it is easier to understand this by running this example
https://github.com/qiaojunfeng/test-julia-OpenAPI/tree/main

@tanmaykm
Copy link
Contributor

tanmaykm commented Jan 3, 2025

Thanks @qiaojunfeng, I think the issue here is missing check for the isBinary flag in some places of the template. Should those be added instead of removing the check altogether? Then the code would be generated as Vector{UInt8} uniformly for binary strings and that should resolve the issue.

@tanmaykm
Copy link
Contributor

tanmaykm commented Jan 3, 2025

@qiaojunfeng Here's my recommended changes: https://github.com/OpenAPITools/openapi-generator/compare/master...tanmaykm:tan/fixfile?expand=1
If it works please feel free to update this PR with that.

@qiaojunfeng
Copy link
Author

@tanmaykm Thanks for the changes!

Indeed using Vector{UInt8} uniformly as the arg type feels cleaner, but unfortunately, it does not work for such cases.

The reason comes from these:

  1. In the client part, the arg file::Vector{UInt8} is used to set the OpenAPI.Clients.Ctx.file, which has type file::Dict{String,String}, so it does not work
function _oacinternal_upload_post(_api::DefaultApi, file::Vector{UInt8}; _mediaType=nothing)
    _ctx = OpenAPI.Clients.Ctx(_api.client, "POST", _returntypes_upload_post_DefaultApi, "/upload", [])
    OpenAPI.Clients.set_param(_ctx.file, "file", file)  # type Vector{UInt8}
    ...

https://github.com/qiaojunfeng/test-julia-OpenAPI/blob/7bed81b4a84fbdb93f2fcaf951e31903b053bb08/client/src/apis/api_DefaultApi.jl#L47-L49
(Your tan/fixfile branch generates the same code as in this link (same as released version of openapi-generator))

  1. When OpenAPI.jl prepares for uploading file, it internally uses curl_mime_filedata, and libcurl c interface expect it to be a filename not file content (https://curl.se/libcurl/c/curl_mime_filedata.html)
            for (_k,_v) in ctx.file
                part = LibCURL.curl_mime_addpart(mime)
                LibCURL.curl_mime_name(part, _k)
                LibCURL.curl_mime_filedata(part, _v)

https://github.com/JuliaComputing/OpenAPI.jl/blob/f18935ed12e40e56d007471e0690216e728e48b0/src/client.jl#L382-L385

Therefore, in my branch, I directly set file as String pointing to the filename, instead of the file content of type Vector{UInt8}

function _oacinternal_upload_post(_api::DefaultApi, file::String; _mediaType=nothing)
    _ctx = OpenAPI.Clients.Ctx(_api.client, "POST", _returntypes_upload_post_DefaultApi, "/upload", [])
    OpenAPI.Clients.set_param(_ctx.file, "file", file)  # type String

https://github.com/qiaojunfeng/test-julia-OpenAPI/blob/7bed81b4a84fbdb93f2fcaf951e31903b053bb08/client_new/src/apis/api_DefaultApi.jl#L47-L49

  1. If we still want to use Vector{UInt8} as the type, probably we need to create a temp file, write the content to it, and pass the temp file name as _v to LibCURL.curl_mime_filedata(part, _v). Since in most cases the file content should be already stored some where as a file, maybe directly passing filename can avoid reading and writing again the content to a temp file, especially useful if the file size is large?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants